{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cggGMzZ8EZ5j"
   },
   "source": [
    "# XLM-R Tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ncsWaH6DEZ5l"
   },
   "source": [
    "In this tutorial, we will train(fine-tune), eval and export a XLM-R model based on a large pre-train model.\n",
    "\n",
    "We will create dummy datasets with a test config such that the model can be trained in a few minutes for demo purpose. To train your model, please update config with your own datasets and tune params in config(e.g. increase epochs)\n",
    "\n",
    "See the full paper and introduction in https://pytext.readthedocs.io/en/master/xlm_r.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xC6QfoNNEZ5m"
   },
   "source": [
    "## Install PyText from source code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "rN2LhQoPF4fO",
    "outputId": "f59679d8-9b2c-4e24-d5b9-89f5cac74d78",
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "# please ignore the warning about tensorboard if it appears\n",
    "!pip install --quiet git+https://github.com/facebookresearch/pytext"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4ZVuq4rKEZ5n"
   },
   "source": [
    "## Download a pre-trained model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7Ca1_O8NEZ5o"
   },
   "source": [
    "We provide 2 pre-trained models \"xlmr.base.v0\" and \"xlmr.large.v0\" at https://pytext.readthedocs.io/en/master/xlm_r.html#pre-trained-models\n",
    "\n",
    "Let's download \"xlmr.large.v0\" and extract the files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "id": "cM-YRbxGEZ5p",
    "outputId": "81e22c0e-6db8-4200-ed5e-52374f1cfa5c",
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "text": [
      "--2020-04-06 16:01:21--  https://dl.fbaipublicfiles.com/fairseq/models/xlmr.large.v0.tar.gz\r\n",
      "Resolving dl.fbaipublicfiles.com (dl.fbaipublicfiles.com)... 2606:4700:10::6816:4a8e, 2606:4700:10::6816:4b8e, 104.22.74.142, ...\r\n",
      "Connecting to dl.fbaipublicfiles.com (dl.fbaipublicfiles.com)|2606:4700:10::6816:4a8e|:443... connected.\r\n",
      "HTTP request sent, awaiting response... 200 OK\r\n",
      "Length: 5116367334 (4.8G) [application/x-tar]\r\n",
      "Saving to: ‘xlmr.large.v0.tar.gz’\r\n",
      "\r\n",
      "xlmr.large.v0.tar.g 100%[===================>]   4.76G  4.57MB/s    in 28m 59s \r\n",
      "\r\n",
      "2020-04-06 16:30:21 (2.81 MB/s) - ‘xlmr.large.v0.tar.gz’ saved [5116367334/5116367334]\r\n",
      "\r\n"
     ],
     "output_type": "stream"
    }
   ],
   "source": [
    "!wget https://dl.fbaipublicfiles.com/fairseq/models/xlmr.large.v0.tar.gz\n",
    "!tar xzf ./xlmr.large.v0.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DnnUY2ooEZ5s"
   },
   "source": [
    "## Create dummy datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 238
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "ZgaasrkkEZ5t",
    "outputId": "70711479-bdcb-4f03-bd97-8b7134c9e1a5",
    "scrolled": true,
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "text": [
      "WARNING:root:This caffe2 python run does not have GPU support. Will run in CPU only mode.\n",
      "/Users/stevenliu/anaconda3/envs/pytext_lib/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
      "/Users/stevenliu/anaconda3/envs/pytext_lib/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
      "/Users/stevenliu/anaconda3/envs/pytext_lib/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
      "/Users/stevenliu/anaconda3/envs/pytext_lib/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
      "/Users/stevenliu/anaconda3/envs/pytext_lib/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
      "/Users/stevenliu/anaconda3/envs/pytext_lib/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n",
      "Install apex from https://github.com/NVIDIA/apex/.\n"
     ],
     "output_type": "stream"
    }
   ],
   "source": [
    "import json\n",
    "import os\n",
    "\n",
    "import torch\n",
    "\n",
    "from pytext import workflow\n",
    "from pytext.config.serialize import pytext_config_from_json\n",
    "from pytext.models.roberta import RoBERTa\n",
    "from pytext.task.serialize import load"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "colab_type": "code",
    "id": "V5CWUupGEZ5x",
    "outputId": "5188c3ef-5248-4614-b371-e626ea96c7fb",
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "text": [
      "Created dummy dataset: dummy_train_file.txt\n",
      "Created dummy dataset: dummy_test_file.txt\n",
      "Created dummy dataset: dummy_eval_file.txt\n"
     ],
     "output_type": "stream"
    }
   ],
   "source": [
    "dummy_train_filename = \"dummy_train_file.txt\"\n",
    "dummy_test_filename = \"dummy_test_file.txt\"\n",
    "dummy_eval_filename = \"dummy_eval_file.txt\"\n",
    "\n",
    "dummy_dataset = \"\"\"neutral\tConceptually cream skimming has two basic dimensions - product and geography.\tProduct and geography are what make cream skimming work. \n",
    "entailment\tyou know during the season and i guess at at your level uh you lose them to the next level if if they decide to recall the the parent team the Braves decide to call to recall a guy from triple A then a double A guy goes up to replace him and a single A guy goes up to replace him\tYou lose the things to the following level if the people recall.\n",
    "entailment\tOne of our number will carry out your instructions minutely.\tA member of my team will execute your orders with immense precision.\n",
    "entailment\tHow do you know? All this is their information again.\tThis information belongs to them.\n",
    "neutral\tyeah i tell you what though if you go price some of those tennis shoes i can see why now you know they're getting up in the hundred dollar range\tThe tennis shoes have a range of prices.\n",
    "entailment\tmy walkman broke so i'm upset now i just have to turn the stereo up real loud\tI'm upset that my walkman broke and now I have to turn the stereo up really loud.\n",
    "neutral\tBut a few Christian mosaics survive above the apse is the Virgin with the infant Jesus, with the Archangel Gabriel to the right (his companion Michael, to the left, has vanished save for a few feathers from his wings).\tMost of the Christian mosaics were destroyed by Muslims.  \n",
    "entailment\t(Read  for Slate 's take on Jackson's findings.)\tSlate had an opinion on Jackson's findings.\n",
    "contradiction\tGays and lesbians.\tHeterosexuals.\n",
    "contradiction\tAt the end of Rue des Francs-Bourgeois is what many consider to be the city's most handsome residential square, the Place des Vosges, with its stone and red brick facades.\tPlace des Vosges is constructed entirely of gray marble.\"\"\"\n",
    "\n",
    "for filename in (dummy_train_filename, dummy_test_filename, dummy_eval_filename):\n",
    "    with open(filename, \"w\") as f:\n",
    "        f.write(dummy_dataset)\n",
    "        print(f\"Created dummy dataset: {filename}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "u1qYtl81EZ51",
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "# To train a real model,set your own dataset here\n",
    "TRAIN_FILENAME = dummy_train_filename\n",
    "TEST_FILENAME = dummy_test_filename\n",
    "EVAL_FILENAME = dummy_eval_filename\n",
    "\n",
    "PRE_TRAIN_MODEL_DIR = \"xlmr.large.v0\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DlVIF4HQzxHo"
   },
   "source": [
    "## Create a PyText Config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "mzbPPx3IEZ54",
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "config_json = \"\"\"\n",
    "{\n",
    "  \"version\": 18,\n",
    "  \"task\": {\n",
    "    \"DocumentClassificationTask\": {\n",
    "      \"data\": {\n",
    "        \"Data\": {\n",
    "          \"source\": {\n",
    "            \"TSVDataSource\": {\n",
    "              \"train_filename\": \"{TRAIN_FILENAME}\",\n",
    "              \"test_filename\": \"{TEST_FILENAME}\",\n",
    "              \"eval_filename\": \"{EVAL_FILENAME}\",\n",
    "              \"field_names\": [\n",
    "                \"label\",\n",
    "                \"text1\",\n",
    "                \"text2\"\n",
    "              ]\n",
    "            }\n",
    "          },\n",
    "          \"batcher\": {\n",
    "            \"Batcher\": {\n",
    "              \"train_batch_size\": 8,\n",
    "              \"eval_batch_size\": 8,\n",
    "              \"test_batch_size\": 8\n",
    "            }\n",
    "          },\n",
    "          \"sort_key\": \"tokens\"\n",
    "        }\n",
    "      },\n",
    "      \"trainer\": {\n",
    "        \"TaskTrainer\": {\n",
    "          \"epochs\": 1,\n",
    "          \"early_stop_after\": 0,\n",
    "          \"max_clip_norm\": null,\n",
    "          \"report_train_metrics\": true,\n",
    "          \"target_time_limit_seconds\": null,\n",
    "          \"do_eval\": true,\n",
    "          \"num_samples_to_log_progress\": 10,\n",
    "          \"num_accumulated_batches\": 1,\n",
    "          \"optimizer\": {\n",
    "            \"Adam\": {\n",
    "              \"lr\": 0.000005,\n",
    "              \"weight_decay\": 0\n",
    "            }\n",
    "          },\n",
    "          \"scheduler\": null,\n",
    "          \"sparsifier\": null,\n",
    "          \"fp16_args\": {\n",
    "            \"FP16OptimizerApex\": {\n",
    "              \"init_loss_scale\": null,\n",
    "              \"min_loss_scale\": null\n",
    "            }\n",
    "          }\n",
    "        }\n",
    "      },\n",
    "      \"model\": {\n",
    "        \"RoBERTa\": {\n",
    "          \"inputs\": {\n",
    "            \"tokens\": {\n",
    "              \"columns\": [\n",
    "                \"text1\",\n",
    "                \"text2\"\n",
    "              ],\n",
    "              \"vocab_file\": \"{VOCAB_PATH}\",\n",
    "              \"tokenizer\": {\n",
    "                \"SentencePieceTokenizer\": {\n",
    "                  \"sp_model_path\": \"{SP_MODEL_PATH}\"\n",
    "                }\n",
    "              },\n",
    "              \"max_seq_len\": 256\n",
    "            },\n",
    "            \"labels\": {\n",
    "              \"LabelTensorizer\": {\n",
    "                \"column\": \"label\",\n",
    "                \"allow_unknown\": false,\n",
    "                \"pad_in_vocab\": false,\n",
    "                \"label_vocab\": null\n",
    "              }\n",
    "            }\n",
    "          },\n",
    "          \"encoder\": {\n",
    "            \"RoBERTaEncoder\": {\n",
    "              \"load_path\": null,\n",
    "              \"save_path\": \"encoder.pt\",\n",
    "              \"shared_module_key\": null,\n",
    "              \"embedding_dim\": 1024,\n",
    "              \"vocab_size\": 250002,\n",
    "              \"num_encoder_layers\": 24,\n",
    "              \"num_attention_heads\": 16,\n",
    "              \"model_path\": \"{PRE_TRAIN_MODEL_PATH}\",\n",
    "              \"is_finetuned\": false\n",
    "            }\n",
    "          },\n",
    "          \"decoder\": {\n",
    "            \"load_path\": null,\n",
    "            \"save_path\": \"decoder.pt\",\n",
    "            \"freeze\": false,\n",
    "            \"shared_module_key\": \"DECODER\",\n",
    "            \"hidden_dims\": [],\n",
    "            \"out_dim\": null,\n",
    "            \"activation\": \"gelu\"\n",
    "          },\n",
    "          \"output_layer\": {\n",
    "            \"load_path\": null,\n",
    "            \"save_path\": null,\n",
    "            \"freeze\": false,\n",
    "            \"shared_module_key\": null,\n",
    "            \"loss\": {\n",
    "              \"CrossEntropyLoss\": {}\n",
    "            },\n",
    "            \"label_weights\": null\n",
    "          }\n",
    "        }\n",
    "      },\n",
    "      \"metric_reporter\": {\n",
    "        \"ClassificationMetricReporter\": {\n",
    "          \"model_select_metric\": \"accuracy\",\n",
    "          \"target_label\": null,\n",
    "          \"text_column_names\": [\n",
    "            \"text1\",\n",
    "            \"text2\"\n",
    "          ],\n",
    "          \"recall_at_precision_thresholds\": []\n",
    "        }\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "ebpc5d0rEZ56",
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "text": [
      "WARNING - Applying old config adapter for version=18. Please consider migrating your old configs to the latest version.\n"
     ],
     "output_type": "stream"
    }
   ],
   "source": [
    "config = pytext_config_from_json(json.loads(config_json))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "o5pjKonYEZ59"
   },
   "source": [
    "## Update Config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "ym70GL87EZ59",
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "config.task.data.source.train_filename = TRAIN_FILENAME\n",
    "config.task.data.source.test_filename = TEST_FILENAME\n",
    "config.task.data.source.eval_filename = EVAL_FILENAME\n",
    "\n",
    "config.task.model.inputs.tokens.tokenizer.sp_model_path = os.path.join(\n",
    "    PRE_TRAIN_MODEL_DIR, \"sentencepiece.bpe.model\"\n",
    ")\n",
    "config.task.model.inputs.tokens.vocab_file = os.path.join(PRE_TRAIN_MODEL_DIR, \"dict.txt\")\n",
    "config.task.model.encoder.model_path = os.path.join(PRE_TRAIN_MODEL_DIR, \"model.pt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YfqEcrxJEZ6A"
   },
   "source": [
    "## Train Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "iDi-XJe4EZ6C",
    "outputId": "fd12ca7d-9809-4229-f94a-a96b50c7bdef",
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "text": [
      "\n",
      "Parameters: PyTextConfig:\n",
      "    auto_resume_from_snapshot: False\n",
      "    debug_path: /tmp/model.debug\n",
      "    distributed_world_size: 1\n",
      "    export_caffe2_path: None\n",
      "    export_onnx_path: /tmp/model.onnx\n",
      "    export_torchscript_path: None\n",
      "    gpu_streams_for_distributed_training: 1\n",
      "    include_dirs: None\n",
      "    load_snapshot_path: \n",
      "    modules_save_dir: \n",
      "    random_seed: None\n",
      "    report_eval_results: False\n",
      "    save_all_checkpoints: False\n",
      "    save_module_checkpoints: False\n",
      "    save_snapshot_path: /tmp/model.pt\n",
      "    task: DocumentClassificationTask.Config:\n",
      "        data: Data.Config:\n",
      "            batcher: Batcher.Config:\n",
      "                eval_batch_size: 8\n",
      "                test_batch_size: 8\n",
      "                train_batch_size: 8\n",
      "            in_memory: True\n",
      "            sort_key: tokens\n",
      "            source: TSVDataSource.Config:\n",
      "                column_mapping: {}\n",
      "                delimiter: \t\n",
      "                drop_incomplete_rows: False\n",
      "                eval_filename: dummy_eval_file.txt\n",
      "                field_names: ['label', 'text1', 'text2']\n",
      "                quoted: False\n",
      "                test_filename: dummy_test_file.txt\n",
      "                train_filename: dummy_train_file.txt\n",
      "        metric_reporter: ClassificationMetricReporter.Config:\n",
      "            additional_column_names: []\n",
      "            model_select_metric: ComparableClassificationMetric.ACCURACY\n",
      "            output_path: /tmp/test_out.txt\n",
      "            pep_format: False\n",
      "            recall_at_precision_thresholds: []\n",
      "            student_column_names: []\n",
      "            target_label: None\n",
      "            text_column_names: ['text1', 'text2']\n",
      "        model: RoBERTa.Config:\n",
      "            decoder: MLPDecoder.Config:\n",
      "                activation: Activation.GELU\n",
      "                dropout: 0.0\n",
      "                freeze: False\n",
      "                hidden_dims: []\n",
      "                layer_norm: False\n",
      "                load_path: None\n",
      "                out_dim: None\n",
      "                save_path: decoder.pt\n",
      "                shared_module_key: DECODER\n",
      "            encoder: RoBERTaEncoder.Config:\n",
      "                embedding_dim: 1024\n",
      "                export: False\n",
      "                freeze: False\n",
      "                is_finetuned: False\n",
      "                load_path: None\n",
      "                model_path: xlmr.large.v0/model.pt\n",
      "                num_attention_heads: 16\n",
      "                num_encoder_layers: 24\n",
      "                output_dropout: 0.4\n",
      "                pooling: PoolingMethod.CLS_TOKEN\n",
      "                save_path: encoder.pt\n",
      "                shared_module_key: None\n",
      "                vocab_size: 250002\n",
      "            inputs: InputConfig:\n",
      "                dense: None\n",
      "                labels: LabelTensorizer.Config:\n",
      "                    allow_unknown: False\n",
      "                    column: label\n",
      "                    is_input: False\n",
      "                    label_vocab: None\n",
      "                    pad_in_vocab: False\n",
      "                tokens: RoBERTaTensorizer.Config:\n",
      "                    base_tokenizer: None\n",
      "                    columns: ['text1', 'text2']\n",
      "                    is_input: True\n",
      "                    max_seq_len: 256\n",
      "                    tokenizer: SentencePieceTokenizer.Config:\n",
      "                        sp_model_path: xlmr.large.v0/sentencepiece.bpe.model\n",
      "                    vocab_file: xlmr.large.v0/dict.txt\n",
      "            output_layer: ClassificationOutputLayer.Config:\n",
      "                freeze: False\n",
      "                label_weights: None\n",
      "                load_path: None\n",
      "                loss: CrossEntropyLoss.Config:\n",
      "                save_path: None\n",
      "                shared_module_key: None\n",
      "        trainer: TaskTrainer.Config:\n",
      "            do_eval: True\n",
      "            early_stop_after: 0\n",
      "            epochs: 1\n",
      "            fp16_args: FP16OptimizerApex.Config:\n",
      "                init_loss_scale: None\n",
      "                min_loss_scale: None\n",
      "                opt_level: O2\n",
      "            load_best_model_after_train: True\n",
      "            max_clip_norm: None\n",
      "            num_accumulated_batches: 1\n",
      "            num_batches_per_epoch: None\n",
      "            num_samples_to_log_progress: 10\n",
      "            optimizer: Adam.Config:\n",
      "                eps: 1e-08\n",
      "                lr: 5e-06\n",
      "                weight_decay: 0.0\n",
      "            report_train_metrics: True\n",
      "            scheduler: None\n",
      "            sparsifier: None\n",
      "            target_time_limit_seconds: None\n",
      "        use_elastic: None\n",
      "    test_out_path: /tmp/test_out.txt\n",
      "    torchscript_quantize: False\n",
      "    use_config_from_snapshot: True\n",
      "    use_cuda_for_testing: True\n",
      "    use_cuda_if_available: True\n",
      "    use_deterministic_cudnn: False\n",
      "    use_fp16: False\n",
      "    use_tensorboard: True\n",
      "    version: 19\n",
      "\n",
      "Cuda is not available, running on CPU...\n",
      "\n",
      "        # for debug of GPU\n",
      "        use_cuda_if_available: True\n",
      "        device_id: 0\n",
      "        world_size: 1\n",
      "        torch.cuda.is_available(): False\n",
      "        cuda.CUDA_ENABLED: False\n",
      "        cuda.DISTRIBUTED_WORLD_SIZE: 1\n",
      "        \n",
      "# for debug of FP16: fp16_enabled=False\n",
      "Creating task: DocumentClassificationTask...\n",
      "PyText data schema: {'text1': <class 'str'>, 'text2': <class 'str'>, 'label': <class 'str'>}.\n",
      "Model :RoBERTa(\n",
      "  (encoder): RoBERTaEncoder(\n",
      "    (output_dropout): Dropout(p=0.4, inplace=False)\n",
      "    (encoder): SentenceEncoder(\n",
      "      (transformer): Transformer(\n",
      "        (token_embedding): Embedding(250002, 1024, padding_idx=1)\n",
      "        (layers): ModuleList(\n",
      "          (0): TransformerLayer(\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "            (attention): MultiheadSelfAttention(\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "              (input_projection): Linear(in_features=1024, out_features=3072, bias=True)\n",
      "              (output_projection): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "            )\n",
      "            (residual_mlp): ResidualMLP(\n",
      "              (mlp): Sequential(\n",
      "                (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "                (1): GeLU()\n",
      "                (2): Dropout(p=0.1, inplace=False)\n",
      "                (3): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "                (4): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (attention_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "            (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "          )\n",
      "          (1): TransformerLayer(\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "            (attention): MultiheadSelfAttention(\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "              (input_projection): Linear(in_features=1024, out_features=3072, bias=True)\n",
      "              (output_projection): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "            )\n",
      "            (residual_mlp): ResidualMLP(\n",
      "              (mlp): Sequential(\n",
      "                (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "                (1): GeLU()\n",
      "                (2): Dropout(p=0.1, inplace=False)\n",
      "                (3): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "                (4): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (attention_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "            (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "          )\n",
      "          (2): TransformerLayer(\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "            (attention): MultiheadSelfAttention(\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "              (input_projection): Linear(in_features=1024, out_features=3072, bias=True)\n",
      "              (output_projection): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "            )\n",
      "            (residual_mlp): ResidualMLP(\n",
      "              (mlp): Sequential(\n",
      "                (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "                (1): GeLU()\n",
      "                (2): Dropout(p=0.1, inplace=False)\n",
      "                (3): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "                (4): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (attention_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "            (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "          )\n",
      "          (3): TransformerLayer(\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "            (attention): MultiheadSelfAttention(\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "              (input_projection): Linear(in_features=1024, out_features=3072, bias=True)\n",
      "              (output_projection): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "            )\n",
      "            (residual_mlp): ResidualMLP(\n",
      "              (mlp): Sequential(\n",
      "                (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "                (1): GeLU()\n",
      "                (2): Dropout(p=0.1, inplace=False)\n",
      "                (3): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "                (4): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (attention_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "            (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "          )\n",
      "          (4): TransformerLayer(\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "            (attention): MultiheadSelfAttention(\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "              (input_projection): Linear(in_features=1024, out_features=3072, bias=True)\n",
      "              (output_projection): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "            )\n",
      "            (residual_mlp): ResidualMLP(\n",
      "              (mlp): Sequential(\n",
      "                (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "                (1): GeLU()\n",
      "                (2): Dropout(p=0.1, inplace=False)\n",
      "                (3): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "                (4): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (attention_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "            (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "          )\n",
      "          (5): TransformerLayer(\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "            (attention): MultiheadSelfAttention(\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "              (input_projection): Linear(in_features=1024, out_features=3072, bias=True)\n",
      "              (output_projection): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "            )\n",
      "            (residual_mlp): ResidualMLP(\n",
      "              (mlp): Sequential(\n",
      "                (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "                (1): GeLU()\n",
      "                (2): Dropout(p=0.1, inplace=False)\n",
      "                (3): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "                (4): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (attention_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "            (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "          )\n",
      "          (6): TransformerLayer(\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "            (attention): MultiheadSelfAttention(\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "              (input_projection): Linear(in_features=1024, out_features=3072, bias=True)\n",
      "              (output_projection): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "            )\n",
      "            (residual_mlp): ResidualMLP(\n",
      "              (mlp): Sequential(\n",
      "                (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "                (1): GeLU()\n",
      "                (2): Dropout(p=0.1, inplace=False)\n",
      "                (3): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "                (4): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (attention_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "            (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "          )\n",
      "          (7): TransformerLayer(\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "            (attention): MultiheadSelfAttention(\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "              (input_projection): Linear(in_features=1024, out_features=3072, bias=True)\n",
      "              (output_projection): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "            )\n",
      "            (residual_mlp): ResidualMLP(\n",
      "              (mlp): Sequential(\n",
      "                (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "                (1): GeLU()\n",
      "                (2): Dropout(p=0.1, inplace=False)\n",
      "                (3): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "                (4): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (attention_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "            (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "          )\n",
      "          (8): TransformerLayer(\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "            (attention): MultiheadSelfAttention(\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "              (input_projection): Linear(in_features=1024, out_features=3072, bias=True)\n",
      "              (output_projection): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "            )\n",
      "            (residual_mlp): ResidualMLP(\n",
      "              (mlp): Sequential(\n",
      "                (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "                (1): GeLU()\n",
      "                (2): Dropout(p=0.1, inplace=False)\n",
      "                (3): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "                (4): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (attention_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "            (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "          )\n",
      "          (9): TransformerLayer(\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "            (attention): MultiheadSelfAttention(\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "              (input_projection): Linear(in_features=1024, out_features=3072, bias=True)\n",
      "              (output_projection): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "            )\n",
      "            (residual_mlp): ResidualMLP(\n",
      "              (mlp): Sequential(\n",
      "                (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "                (1): GeLU()\n",
      "                (2): Dropout(p=0.1, inplace=False)\n",
      "                (3): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "                (4): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (attention_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "            (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "          )\n",
      "          (10): TransformerLayer(\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "            (attention): MultiheadSelfAttention(\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "              (input_projection): Linear(in_features=1024, out_features=3072, bias=True)\n",
      "              (output_projection): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "            )\n",
      "            (residual_mlp): ResidualMLP(\n",
      "              (mlp): Sequential(\n",
      "                (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "                (1): GeLU()\n",
      "                (2): Dropout(p=0.1, inplace=False)\n",
      "                (3): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "                (4): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (attention_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "            (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "          )\n",
      "          (11): TransformerLayer(\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "            (attention): MultiheadSelfAttention(\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "              (input_projection): Linear(in_features=1024, out_features=3072, bias=True)\n",
      "              (output_projection): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "            )\n",
      "            (residual_mlp): ResidualMLP(\n",
      "              (mlp): Sequential(\n",
      "                (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "                (1): GeLU()\n",
      "                (2): Dropout(p=0.1, inplace=False)\n",
      "                (3): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "                (4): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (attention_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "            (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "          )\n",
      "          (12): TransformerLayer(\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "            (attention): MultiheadSelfAttention(\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "              (input_projection): Linear(in_features=1024, out_features=3072, bias=True)\n",
      "              (output_projection): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "            )\n",
      "            (residual_mlp): ResidualMLP(\n",
      "              (mlp): Sequential(\n",
      "                (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "                (1): GeLU()\n",
      "                (2): Dropout(p=0.1, inplace=False)\n",
      "                (3): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "                (4): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (attention_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "            (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "          )\n",
      "          (13): TransformerLayer(\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "            (attention): MultiheadSelfAttention(\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "              (input_projection): Linear(in_features=1024, out_features=3072, bias=True)\n",
      "              (output_projection): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "            )\n",
      "            (residual_mlp): ResidualMLP(\n",
      "              (mlp): Sequential(\n",
      "                (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "                (1): GeLU()\n",
      "                (2): Dropout(p=0.1, inplace=False)\n",
      "                (3): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "                (4): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (attention_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "            (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "          )\n",
      "          (14): TransformerLayer(\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "            (attention): MultiheadSelfAttention(\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "              (input_projection): Linear(in_features=1024, out_features=3072, bias=True)\n",
      "              (output_projection): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "            )\n",
      "            (residual_mlp): ResidualMLP(\n",
      "              (mlp): Sequential(\n",
      "                (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "                (1): GeLU()\n",
      "                (2): Dropout(p=0.1, inplace=False)\n",
      "                (3): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "                (4): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (attention_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "            (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "          )\n",
      "          (15): TransformerLayer(\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "            (attention): MultiheadSelfAttention(\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "              (input_projection): Linear(in_features=1024, out_features=3072, bias=True)\n",
      "              (output_projection): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "            )\n",
      "            (residual_mlp): ResidualMLP(\n",
      "              (mlp): Sequential(\n",
      "                (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "                (1): GeLU()\n",
      "                (2): Dropout(p=0.1, inplace=False)\n",
      "                (3): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "                (4): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (attention_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "            (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "          )\n",
      "          (16): TransformerLayer(\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "            (attention): MultiheadSelfAttention(\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "              (input_projection): Linear(in_features=1024, out_features=3072, bias=True)\n",
      "              (output_projection): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "            )\n",
      "            (residual_mlp): ResidualMLP(\n",
      "              (mlp): Sequential(\n",
      "                (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "                (1): GeLU()\n",
      "                (2): Dropout(p=0.1, inplace=False)\n",
      "                (3): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "                (4): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (attention_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "            (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "          )\n",
      "          (17): TransformerLayer(\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "            (attention): MultiheadSelfAttention(\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "              (input_projection): Linear(in_features=1024, out_features=3072, bias=True)\n",
      "              (output_projection): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "            )\n",
      "            (residual_mlp): ResidualMLP(\n",
      "              (mlp): Sequential(\n",
      "                (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "                (1): GeLU()\n",
      "                (2): Dropout(p=0.1, inplace=False)\n",
      "                (3): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "                (4): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (attention_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "            (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "          )\n",
      "          (18): TransformerLayer(\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "            (attention): MultiheadSelfAttention(\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "              (input_projection): Linear(in_features=1024, out_features=3072, bias=True)\n",
      "              (output_projection): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "            )\n",
      "            (residual_mlp): ResidualMLP(\n",
      "              (mlp): Sequential(\n",
      "                (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "                (1): GeLU()\n",
      "                (2): Dropout(p=0.1, inplace=False)\n",
      "                (3): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "                (4): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (attention_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "            (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "          )\n",
      "          (19): TransformerLayer(\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "            (attention): MultiheadSelfAttention(\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "              (input_projection): Linear(in_features=1024, out_features=3072, bias=True)\n",
      "              (output_projection): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "            )\n",
      "            (residual_mlp): ResidualMLP(\n",
      "              (mlp): Sequential(\n",
      "                (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "                (1): GeLU()\n",
      "                (2): Dropout(p=0.1, inplace=False)\n",
      "                (3): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "                (4): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (attention_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "            (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "          )\n",
      "          (20): TransformerLayer(\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "            (attention): MultiheadSelfAttention(\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "              (input_projection): Linear(in_features=1024, out_features=3072, bias=True)\n",
      "              (output_projection): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "            )\n",
      "            (residual_mlp): ResidualMLP(\n",
      "              (mlp): Sequential(\n",
      "                (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "                (1): GeLU()\n",
      "                (2): Dropout(p=0.1, inplace=False)\n",
      "                (3): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "                (4): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (attention_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "            (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "          )\n",
      "          (21): TransformerLayer(\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "            (attention): MultiheadSelfAttention(\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "              (input_projection): Linear(in_features=1024, out_features=3072, bias=True)\n",
      "              (output_projection): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "            )\n",
      "            (residual_mlp): ResidualMLP(\n",
      "              (mlp): Sequential(\n",
      "                (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "                (1): GeLU()\n",
      "                (2): Dropout(p=0.1, inplace=False)\n",
      "                (3): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "                (4): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (attention_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "            (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "          )\n",
      "          (22): TransformerLayer(\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "            (attention): MultiheadSelfAttention(\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "              (input_projection): Linear(in_features=1024, out_features=3072, bias=True)\n",
      "              (output_projection): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "            )\n",
      "            (residual_mlp): ResidualMLP(\n",
      "              (mlp): Sequential(\n",
      "                (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "                (1): GeLU()\n",
      "                (2): Dropout(p=0.1, inplace=False)\n",
      "                (3): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "                (4): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (attention_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "            (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "          )\n",
      "          (23): TransformerLayer(\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "            (attention): MultiheadSelfAttention(\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "              (input_projection): Linear(in_features=1024, out_features=3072, bias=True)\n",
      "              (output_projection): Linear(in_features=1024, out_features=1024, bias=True)\n",
      "            )\n",
      "            (residual_mlp): ResidualMLP(\n",
      "              (mlp): Sequential(\n",
      "                (0): Linear(in_features=1024, out_features=4096, bias=True)\n",
      "                (1): GeLU()\n",
      "                (2): Dropout(p=0.1, inplace=False)\n",
      "                (3): Linear(in_features=4096, out_features=1024, bias=True)\n",
      "                (4): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (attention_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "            (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "          )\n",
      "        )\n",
      "        (positional_embedding): PositionalEmbedding(\n",
      "          (embedding): Embedding(514, 1024, padding_idx=1)\n",
      "        )\n",
      "        (embedding_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)\n",
      "        (dropout): Dropout(p=0.1, inplace=False)\n",
      "      )\n",
      "    )\n",
      "  )\n",
      "  (decoder): MLPDecoder(\n",
      "    (mlp): Sequential(\n",
      "      (0): Linear(in_features=1024, out_features=3, bias=True)\n",
      "    )\n",
      "  )\n",
      "  (output_layer): MulticlassOutputLayer()\n",
      ")\n",
      "Num trainable parameters: 558842883\n",
      "\n",
      "Worker 0 starting epoch 1\n",
      "Learning rate(s): 5e-06\n",
      "start training epoch 1\n",
      "\n",
      "\n",
      "Stage.TRAIN\n",
      "Epoch:1\n",
      "loss: 1.045339\n",
      "Accuracy: 30.00\n",
      "\n",
      "Soft Metrics:\n",
      "+---------------+-------------------+---------+\n",
      "| Label         | Average precision | ROC AUC |\n",
      "+---------------+-------------------+---------+\n",
      "| contradiction |             0.333 |   0.625 |\n",
      "|    entailment |             0.642 |   0.680 |\n",
      "|       neutral |             0.667 |   0.667 |\n",
      "+---------------+-------------------+---------+\n",
      "\n",
      "Recall at Precision\n",
      "+---------------+\n",
      "| Label         |\n",
      "+---------------+\n",
      "| contradiction |\n",
      "| entailment    |\n",
      "| neutral       |\n",
      "+---------------+\n",
      "\n",
      "Precision at Recall\n",
      "+---------------+---------+---------+---------+---------+---------+\n",
      "| Label         | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |\n",
      "+---------------+---------+---------+---------+---------+---------+\n",
      "| contradiction |   0.333 |   0.333 |   0.333 |   0.333 |   0.333 |\n",
      "| entailment    |   0.750 |   0.750 |   0.750 |   0.667 |   0.625 |\n",
      "| neutral       |   1.000 |   0.667 |   0.667 |   0.333 |   0.333 |\n",
      "+---------------+---------+---------+---------+---------+---------+\n",
      "+------------------------------------------------------------------------------------+---------+---------+--------+---------+--------+--------+-------+\n",
      "| Stage                                                                              | Total   | Average | Max    | P50     | P90    | P99    | Count |\n",
      "+------------------------------------------------------------------------------------+---------+---------+--------+---------+--------+--------+-------+\n",
      "| Trainer.train_from_state -> train epoch -> report metrics                          |  18.5ms |  18.5ms | 18.5ms |  18.5ms | 18.5ms | 18.5ms |     1 |\n",
      "| Trainer.train_from_state -> train epoch -> run_step                                |   51.9s |   26.0s |  26.4s |   26.0s |  26.3s |  26.3s |     2 |\n",
      "| Trainer.train_from_state -> train epoch -> run_step -> add metrics                 |   3.6ms |   1.8ms |  2.7ms |   1.8ms |  2.5ms |  2.6ms |     2 |\n",
      "| Trainer.train_from_state -> train epoch -> run_step -> backprop                    |   10.7s |    5.4s |   6.1s |    5.4s |   6.0s |   6.1s |     2 |\n",
      "| Trainer.train_from_state -> train epoch -> run_step -> backprop -> loss.backward   |   10.7s |    5.4s |   6.1s |    5.4s |   6.0s |   6.1s |     2 |\n",
      "| Trainer.train_from_state -> train epoch -> run_step -> model.train_batch           |    6.7s |    3.3s |   5.0s |    3.3s |   4.6s |   4.9s |     2 |\n",
      "| Trainer.train_from_state -> train epoch -> run_step -> optimizer                   |   32.7s |   16.4s |  17.5s |   16.4s |  17.2s |  17.4s |     2 |\n",
      "| Trainer.train_from_state -> train epoch -> run_step -> optimizer -> optimizer.step |   32.7s |   16.4s |  17.5s |   16.4s |  17.2s |  17.4s |     2 |\n",
      "| Trainer.train_from_state -> train epoch -> run_step -> sparsifier                  | 100.3ns |  50.2ns | 51.4ns |  50.2ns | 51.1ns | 51.3ns |     2 |\n",
      "| Trainer.train_from_state -> train epoch -> run_step -> zero gradients              |    1.7s | 861.1ms |   1.7s | 861.1ms |   1.5s |   1.7s |     2 |\n",
      "+------------------------------------------------------------------------------------+---------+---------+--------+---------+--------+--------+-------+\n",
      "| Total time                                                                         | 52.6s   |         |        |         |        |        |       |\n",
      "+------------------------------------------------------------------------------------+---------+---------+--------+---------+--------+--------+-------+\n",
      "start evaluating epoch 1\n",
      "\n",
      "\n",
      "Stage.EVAL\n",
      "Epoch:1\n",
      "loss: 1.186789\n",
      "Accuracy: 30.00\n",
      "\n",
      "Soft Metrics:\n",
      "+---------------+-------------------+---------+\n",
      "| Label         | Average precision | ROC AUC |\n",
      "+---------------+-------------------+---------+\n",
      "| contradiction |             0.833 |   0.938 |\n",
      "|    entailment |             0.668 |   0.720 |\n",
      "|       neutral |             0.656 |   0.619 |\n",
      "+---------------+-------------------+---------+\n",
      "\n",
      "Recall at Precision\n",
      "+---------------+\n",
      "| Label         |\n",
      "+---------------+\n",
      "| contradiction |\n",
      "| entailment    |\n",
      "| neutral       |\n",
      "+---------------+\n",
      "\n",
      "Precision at Recall\n",
      "+---------------+---------+---------+---------+---------+---------+\n",
      "| Label         | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |\n",
      "+---------------+---------+---------+---------+---------+---------+\n",
      "| contradiction |   1.000 |   1.000 |   0.667 |   0.667 |   0.667 |\n",
      "| entailment    |   0.800 |   0.800 |   0.800 |   0.800 |   0.625 |\n",
      "| neutral       |   1.000 |   0.667 |   0.667 |   0.300 |   0.300 |\n",
      "+---------------+---------+---------+---------+---------+---------+\n",
      "+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+\n",
      "| Stage                                                                   | Total   | Average | Max     | P50     | P90     | P99     | Count |\n",
      "+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+\n",
      "| Trainer.train_from_state -> eval epoch -> report metrics                |   1.9ms |   1.9ms |   1.9ms |   1.9ms |   1.9ms |   1.9ms |     1 |\n",
      "| Trainer.train_from_state -> eval epoch -> run_step                      |    3.6s |    1.8s |    2.9s |    1.8s |    2.7s |    2.9s |     2 |\n",
      "| Trainer.train_from_state -> eval epoch -> run_step -> add metrics       | 422.6ns | 211.3ns | 333.0ns | 211.3ns | 308.7ns | 330.6ns |     2 |\n",
      "| Trainer.train_from_state -> eval epoch -> run_step -> backprop          |  13.2ns |   6.6ns |   7.5ns |   6.6ns |   7.3ns |   7.5ns |     2 |\n",
      "| Trainer.train_from_state -> eval epoch -> run_step -> model.train_batch |    3.6s |    1.8s |    2.9s |    1.8s |    2.7s |    2.9s |     2 |\n",
      "| Trainer.train_from_state -> eval epoch -> run_step -> optimizer         |  18.3ns |   9.2ns |  12.7ns |   9.2ns |  12.0ns |  12.6ns |     2 |\n",
      "| Trainer.train_from_state -> eval epoch -> run_step -> sparsifier        |  15.8ns |   7.9ns |  11.8ns |   7.9ns |  11.0ns |  11.7ns |     2 |\n",
      "| Trainer.train_from_state -> eval epoch -> run_step -> zero gradients    |  34.0ns |  17.0ns |  29.3ns |  17.0ns |  26.9ns |  29.1ns |     2 |\n",
      "+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+\n",
      "| Total time                                                              | 3.6s    |         |         |         |         |         |       |\n",
      "+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+\n",
      "Found a better model!\n",
      "Saving state of module RoBERTaEncoder to encoder.pt ...\n",
      "Saving state of module MLPDecoder to decoder.pt ...\n",
      "\n",
      "=== Saving model to: /tmp/model.pt\n",
      "Saving pytorch model to: /tmp/model.pt\n",
      "Training timings\n",
      "+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+\n",
      "| Stage                                                                              | Total   | Average | Max     | P50     | P90     | P99     | Count |\n",
      "+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+\n",
      "| Trainer.train_from_state                                                           |   1m18s |   1m18s |   1m18s |   1m18s |   1m18s |   1m18s |     1 |\n",
      "| Trainer.train_from_state -> eval epoch                                             |    3.6s |    3.6s |    3.6s |    3.6s |    3.6s |    3.6s |     1 |\n",
      "| Trainer.train_from_state -> eval epoch -> report metrics                           |   1.9ms |   1.9ms |   1.9ms |   1.9ms |   1.9ms |   1.9ms |     1 |\n",
      "| Trainer.train_from_state -> eval epoch -> run_step                                 |    3.6s |    1.8s |    2.9s |    1.8s |    2.7s |    2.9s |     2 |\n",
      "| Trainer.train_from_state -> eval epoch -> run_step -> add metrics                  | 422.6ns | 211.3ns | 333.0ns | 211.3ns | 308.7ns | 330.6ns |     2 |\n",
      "| Trainer.train_from_state -> eval epoch -> run_step -> backprop                     |  13.2ns |   6.6ns |   7.5ns |   6.6ns |   7.3ns |   7.5ns |     2 |\n",
      "| Trainer.train_from_state -> eval epoch -> run_step -> model.train_batch            |    3.6s |    1.8s |    2.9s |    1.8s |    2.7s |    2.9s |     2 |\n",
      "| Trainer.train_from_state -> eval epoch -> run_step -> optimizer                    |  18.3ns |   9.2ns |  12.7ns |   9.2ns |  12.0ns |  12.6ns |     2 |\n",
      "| Trainer.train_from_state -> eval epoch -> run_step -> sparsifier                   |  15.8ns |   7.9ns |  11.8ns |   7.9ns |  11.0ns |  11.7ns |     2 |\n",
      "| Trainer.train_from_state -> eval epoch -> run_step -> zero gradients               |  34.0ns |  17.0ns |  29.3ns |  17.0ns |  26.9ns |  29.1ns |     2 |\n",
      "| Trainer.train_from_state -> pre-training                                           |  90.8ns |  90.8ns |  90.8ns |  90.8ns |  90.8ns |  90.8ns |     1 |\n",
      "| Trainer.train_from_state -> save checkpoint                                        |   19.4s |   19.4s |   19.4s |   19.4s |   19.4s |   19.4s |     1 |\n",
      "| Trainer.train_from_state -> train epoch                                            |   52.6s |   52.6s |   52.6s |   52.6s |   52.6s |   52.6s |     1 |\n",
      "| Trainer.train_from_state -> train epoch -> report metrics                          |  18.5ms |  18.5ms |  18.5ms |  18.5ms |  18.5ms |  18.5ms |     1 |\n",
      "| Trainer.train_from_state -> train epoch -> run_step                                |   51.9s |   26.0s |   26.4s |   26.0s |   26.3s |   26.3s |     2 |\n",
      "| Trainer.train_from_state -> train epoch -> run_step -> add metrics                 |   3.6ms |   1.8ms |   2.7ms |   1.8ms |   2.5ms |   2.6ms |     2 |\n",
      "| Trainer.train_from_state -> train epoch -> run_step -> backprop                    |   10.7s |    5.4s |    6.1s |    5.4s |    6.0s |    6.1s |     2 |\n",
      "| Trainer.train_from_state -> train epoch -> run_step -> backprop -> loss.backward   |   10.7s |    5.4s |    6.1s |    5.4s |    6.0s |    6.1s |     2 |\n",
      "| Trainer.train_from_state -> train epoch -> run_step -> model.train_batch           |    6.7s |    3.3s |    5.0s |    3.3s |    4.6s |    4.9s |     2 |\n",
      "| Trainer.train_from_state -> train epoch -> run_step -> optimizer                   |   32.7s |   16.4s |   17.5s |   16.4s |   17.2s |   17.4s |     2 |\n",
      "| Trainer.train_from_state -> train epoch -> run_step -> optimizer -> optimizer.step |   32.7s |   16.4s |   17.5s |   16.4s |   17.2s |   17.4s |     2 |\n",
      "| Trainer.train_from_state -> train epoch -> run_step -> sparsifier                  | 100.3ns |  50.2ns |  51.4ns |  50.2ns |  51.1ns |  51.3ns |     2 |\n",
      "| Trainer.train_from_state -> train epoch -> run_step -> zero gradients              |    1.7s | 861.1ms |    1.7s | 861.1ms |    1.5s |    1.7s |     2 |\n",
      "+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+\n",
      "| Total time                                                                         | 2m3s    |         |         |         |         |         |       |\n",
      "+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+\n",
      "Destroying TSV object\n",
      "Total number of rows read: 0\n",
      "Total number of rows dropped: 0\n",
      "Destroying TSV object\n",
      "Total number of rows read: 0\n",
      "Total number of rows dropped: 0\n",
      "Destroying TSV object\n",
      "Total number of rows read: 0\n",
      "Total number of rows dropped: 0\n"
     ],
     "output_type": "stream"
    }
   ],
   "source": [
    "trained_model, best_metric = workflow.train_model(config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1ix1ebtfEZ6E"
   },
   "source": [
    "## Load Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "id": "YsPoVJA3EZ6F",
    "outputId": "975067c8-53af-4446-bc69-74b7a8ae8859",
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "text": [
      "Loaded checkpoint...\n",
      "Use config saved in snapshot\n",
      "Creating task: DocumentClassificationTask...\n",
      "PyText data schema: {'text1': <class 'str'>, 'text2': <class 'str'>, 'label': <class 'str'>}.\n",
      "Skipped initializing tensorizers since they are loaded from a previously saved state.\n",
      "Loading model from model state dict...\n",
      "Loaded!\n"
     ],
     "output_type": "stream"
    }
   ],
   "source": [
    "model_file = \"/tmp/model.pt\"\n",
    "task, config, _ = load(model_file)\n",
    "loaded_model = task.model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "1MUXFZxfEZ6H",
    "outputId": "c9c89a1b-69d0-4461-f680-12da4a004456",
    "pycharm": {
     "is_executing": false,
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": "True"
     },
     "metadata": {},
     "output_type": "execute_result",
     "execution_count": 11
    }
   ],
   "source": [
    "# loaded_model is a torch.nn.Module\n",
    "isinstance(loaded_model, torch.nn.Module)\n"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "bento_stylesheets": {
   "bento/extensions/flow/main.css": true,
   "bento/extensions/kernel_selector/main.css": true,
   "bento/extensions/kernel_ui/main.css": true,
   "bento/extensions/new_kernel/main.css": true,
   "bento/extensions/system_usage/main.css": true,
   "bento/extensions/theme/main.css": true
  },
  "colab": {
   "collapsed_sections": [],
   "name": "Copy of xlm_r_tutorial.ipynb",
   "provenance": []
  },
  "disseminate_notebook_id": {
   "notebook_id": "3080270822000663"
  },
  "disseminate_notebook_info": {
   "bento_version": "20191118-000256",
   "description": "Tutorial to train a XLM-RoBERTa model",
   "hide_code": false,
   "hipster_group": "",
   "kernel_build_info": {
    "deps": [
     "//pytext:main_lib",
     "//pytext/fb:main_lib",
     "//pytext/fb/pur:pur_lib",
     "//fblearner/flow/projects/langtech/pytext_model:types",
     "//configerator/structs/sigrid:model_id-py",
     "//fblearner/predictor/clients:client_lib",
     "//fblearner/predictor/clients/py:api",
     "//fblearner/predictor/clients/tests:model_test_fixtures",
     "//fblearner/predictor/clients/py3:client"
    ],
    "external_deps": []
   },
   "no_uii": true,
   "notebook_number": "176995",
   "others_can_edit": false,
   "reviewers": "",
   "revision_id": "552431058491293",
   "tags": "",
   "tasks": "",
   "title": "XLM-R Tutorial"
  },
  "kernelspec": {
   "name": "python3",
   "language": "python",
   "display_name": "Python 3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5+"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "source": [],
    "metadata": {
     "collapsed": false
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}